Sorites Model Results

Model fit: Final Design

Old version

In Noah’s slides, we show model fit for only the final prior elicitaiton and the final sorites design (relative clause, with the final set of ranges). In the past, we fit the price prior parameters in R and then did a course grid search for speakerOptimality. We used the joint posterior distribution over the price \(x\) and the threshold \(\theta\) from listener1 (the pragmatic listener) to get model predictions for both sorites premises. For inductive, we took the proportion of the model samples where \(x-\varepsilon > \theta\). For concrete, we took the proportion of the model samples where \(given\ price > \theta\). We z-scored participants’ responses for the correlation plots.

BDA version

We can reproduce the old results from Noah’s slides with our new webppl implementation with Bayesiean data analysis. We can simultaneously fit participants’ bin ratings from the final prior elicitation and their responses for the final version of the sorites premises study.

Fitting to the final sorites and bins data does not significantly change the inferred priors relative to fitting only to the bins data. That is, the prior fits are being driven by the bins data and the concrete and inductive premise data mostly affect the optimality parameter. Since this is exactly how I used that information in the old version, it makes sense that we reproduce the results even with the BDA.

If we only give the model data from the final bins prior elicitation, the model has high uncertainty about the concrete premise, but has a decent correlation with the data.

However, the priors on prices we would infer from the sorites responses alone are quite different from both the give-a-number priors and the final experiment bins priors.

In this situation, when we just fit the priors to the sorites data, the model can “explain” a lot of the variance, but it’s likely overfitting to this particular dataset. I haven’t tried to predict held-out data, which would show how much overfitting is happening here.

Model fit: All Designs

Fit to all data, prior and sorites

Fit to all prior data, no sorites

Fit to all sorites data, no prior data

Discretization

As discussed in data_summary.html, there were different prices used in the designs for different experiments. Since the range of prices is so wide, and since some prices are very close together, it would be difficult to create a discretization that works for every dollar amount used across all experiments. Because of this, I have implemented a separate discretization for each experiment.

The discretization works by creating a set of bins (not necessarily of equal width) such that each price in the experiment falls into a unique bin. There must also be bins below the lowest price’s bin and above the highest price’s bin. The probability of each bin is computed from the cdf: prob_bin = pnorm(upper_boundary) - pnorm(lower_boundary). The probability of theta falling between prices from adjacent bins depends on the widths of those bins (and also the cdf, but I’m ignoring that for now).

Here’s the discretization used for the full set of sorites experiments.

Bins are shown divided by vertical lines. The midpoints of the bins, used as the value, appear in black. The thetas, which appear at the boundaries of the bins, are shown in grey. The height of the bars represent the probability that theta will fall between each pair of midpoints.

Likert responses

For a linking function from S1 endorsement probabilities to Likert scale responses, I use a binomial with parameter equal to the endorsement probability. So for endorsement probability 0.7, the expected distribution of Likert responses would be